Large Language Models (LLMs) have achieved remarkable success in software engineering tasks when trained with executable runtime environments, particularly in resolving GitHub issues. However, such runtime environments are often unavailable in other domains, especially cybersecurity, where challenge configurations and execution contexts are ephemeral or restricted. We present Cyber-Zero, the first runtime-free framework for synthesizing high-quality agent trajectories to train cybersecurity LLMs. Cyber-Zero leverages publicly available CTF writeups and employs persona-driven LLM simulation to reverse-engineer runtime behaviors and generate realistic, long-horizon interaction sequences without actual environments. Using trajectories synthesized by Cyber-Zero, we train LLM-based agents that achieve up to 13.1% absolute performance gains over baseline models on three prominent CTF benchmarks: InterCode-CTF, NYU CTF Bench, and Cybench. Our best model, Cyber-Zero-32B, establishes new state-of-the-art performance among open-weight models, matching the capabilities of proprietary systems like DeepSeek-V3-0324 and Claude-3.5-Sonnet while offering superior cost-effectiveness, and demonstrating that runtime-free trajectory synthesis can effectively democratize the development of state-of-the-art cybersecurity agents.

Artificial Intelligence (AI) can be a force for good in our future, that much is obvious from the fact that it's being utilized to advance things like medical research. The thought that somewhere out there, there's a James Bond-like villain in an armchair stroking a cat and using generative AI to hack your PC may seem like fantasy but, quite frankly, it's not. Cyber security experts are already scrambling to thwart millions of threats by hackers that have used generative AI to hack PCs, steal money, credentials, and data, and, with the rapid proliferation of new and improved AI tools, it's only going to get worse. The type of cyberattacks hackers are using aren't necessarily new. They're just more prolific, sophisticated, and effective now that they have weaponized AI.

Generative AI agents, software systems powered by Large Language Models (LLMs), are emerging as a promising approach to automate cybersecurity tasks. Among the others, penetration testing is a challenging field due to the task complexity and the diverse strategies to simulate cyber-attacks. Despite growing interest and initial studies in automating penetration testing with generative agents, there remains a significant gap in the form of a comprehensive and standard framework for their evaluation and development. This paper introduces AutoPenBench, an open benchmark for evaluating generative agents in automated penetration testing. We present a comprehensive framework that includes 33 tasks, each representing a vulnerable system that the agent has to attack. Tasks are of increasing difficulty levels, including in-vitro and real-world scenarios. We assess the agent performance with generic and specific milestones that allow us to compare results in a standardised manner and understand the limits of the agent under test. We show the benefits of AutoPenBench by testing two agent architectures: a fully autonomous and a semi-autonomous supporting human interaction. We compare their performance and limitations. For example, the fully autonomous agent performs unsatisfactorily achieving a 21% Success Rate (SR) across the benchmark, solving 27% of the simple tasks and only one real-world task. In contrast, the assisted agent demonstrates substantial improvements, with 64% of SR. AutoPenBench allows us also to observe how different LLMs like GPT-4o or OpenAI o1 impact the ability of the agents to complete the tasks. We believe that our benchmark fills the gap with a standard and flexible framework to compare penetration testing agents on a common ground. We hope to extend AutoPenBench along with the research community by making it available under https://github.com/lucagioacchini/auto-pen-bench.

Our approach to time series anomaly detection is computationally efficient, automatically learns how to update probabilities and adapt to changes in data. As we describe in the next section, this approach has yielded successful attack detection at high precision. The proposed time series anomaly detection model was deployed and utilized by Microsoft Threat Experts to detect RDP brute force attacks during threat hunting activities. A list that ranks machines across enterprises with the lowest anomaly scores (indicating the likelihood of observing a value at least as large under expected conditions in all signals considered) is updated and reviewed every day. See Table 1 for an example.

In this special guest feature, Sekhar Sarukkai, Chief Scientist at Skyhigh Networks, discusses the power of machine learning and user behavior analytics in detecting and mitigating the effects of cyberattacks before financial loss occurs. Sekhar is responsible for driving innovations in cloud security technology. He brings more than 20 years of experience in enterprise networking, security, and cloud services development. Prior to founding Skyhigh Networks, Sekhar was a Sr. Director of Engineering at Cisco Systems responsible for delivering Cisco's market leading network access control products, including Cisco's Identity Services Engine. He started his career at NASA Ames Research Center after obtaining his MS/PhD in Computer Science from Indiana University.